Stabilizing video using transformation matrices

ABSTRACT

In general, the subject matter can be embodied in methods, systems, and program products for identifying, by a computing system and using first and second frames of a video, a transformation that indicates movement of a camera with respect to the frames. The computing system generates a modified transformation so that the transformation is less representative of recent movement. The computing system uses the transformation and the modified transformation to generate a second transformation. The computing system identifies an anticipated distortion that would be present in a stabilized version of the second frame. The computing system determines an amount by which to reduce a stabilizing effect. The computing system applies the second transformation to the second frame to stabilize the second frame, where the stabilizing effect has been reduced based on the determined amount by which to reduce the stabilizing effect.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S.application Ser. No. 14/883,515, filed on Oct. 14, 2015, the entirecontents of which are hereby incorporated by reference.

TECHNICAL FIELD

This document generally relates to stabilizing video.

BACKGROUND

Video recording used to be the domain of dedicated video recordingdevices, but it is more common to find everyday devices such as cellulartelephones and tablet computers that are able to record video. An issuewith most handheld recording devices is that these devices suffer fromvideo shake, in which a user's involuntary movements while holding therecording device affects a quality of the video.

Shaking the recording device can result in an equally-shaky video unlessthat shaking is compensated, for example, by a video stabilizationmechanism. Optical video stabilization can decrease the shaking presentin video by mechanically moving components of the recording device, suchas a lens or the image sensor. Optical video stabilizing devices,however, may add to the material and manufacturing costs of a recordingdevice. Moreover, optical video stabilization devices may add to thesize of a recording device, and there is often a desire to designrecording devices to be small.

SUMMARY

This document describes techniques, methods, systems, and othermechanisms for stabilizing video.

As additional description to the embodiments described below, thepresent disclosure describes the following embodiments.

Embodiment 1 is a computer-implemented method. The method comprisesreceiving, by a computing system, first and second frames of a videothat was captured by a recording device. The method comprisesidentifying, by the computing system and using the first and secondframes of the video, a mathematical transformation that indicatesmovement of the camera with respect to a scene captured by the videofrom when the first frame was captured to when the second frame wascaptured. The method comprises generating, by the computing system, amodified mathematical transformation by modifying the mathematicaltransformation that indicates the movement of the camera with respect tothe scene, so that the mathematical transformation is lessrepresentative of movement that began recently. The method comprisesgenerating, by the computing system using the mathematicaltransformation and the modified mathematical transformation, a secondmathematical transformation that is able to be applied to the secondframe to stabilize the second frame. The method comprises identifying,by the computing system, an anticipated distortion that would be presentin a stabilized version of the second frame resulting from applicationof the second mathematical transformation to the second frame, based ona difference between: (i) an amount of distortion in the horizontaldirection resulting from application of the second mathematicaltransformation to the second frame, and (ii) an amount of distortion inthe vertical direction resulting from application of the secondmathematical transformation to the second frame. The method comprisesdetermining, by the computing system, an amount by which to reduce astabilizing effect that results from application of the secondmathematical transformation to the second frame, based on a degree towhich the anticipated distortion exceeds an acceptable change indistortion that was calculated from distortion in multiple frames of thevideo that preceded the second frame. The method comprises generating,by the computing system, the stabilized version of the second frame byapplying the second mathematical transformation to the second frame,where a stabilizing effect of applying the second mathematicaltransformation to the second frame has been reduced based on thedetermined amount by which to reduce the stabilizing effect.

Embodiment 2 is the method of embodiment 1, wherein the second frame isa frame of the video that immediately follows the first frame of thevideo.

Embodiment 3 is the method of embodiment 1, wherein the mathematicaltransformation that indicates movement of the camera includes ahomography transform matrix.

Embodiment 4 is the method of embodiment 3, wherein modifying themathematical transformation includes applying a lowpass filter to thehomography transform matrix.

Embodiment 5 is the method of embodiment 3, wherein the anticipateddistortion is based on a difference between a horizontal zoom value inthe second mathematical transformation and a vertical zoom value in thesecond mathematical transformation.

Embodiment 6 is the method of embodiment 1, wherein modifying themathematical transformation includes modifying the mathematicaltransformation so that the modified mathematical transformation is morerepresentative of movement that has been occurring over a long period oftime than the mathematical transformation.

Embodiment 7 is the method of embodiment 1, wherein determining theamount by which to reduce the stabilizing effect that results fromapplication of the second mathematical transformation to the secondframe is further based on a determined speed of movement of the camerafrom the first frame to the second frame exceeding an acceptable changein speed of movement of the camera that was calculated based on a speedof movement of the camera between multiple frames of the video thatpreceded the second frame.

Embodiment 8 is the method of embodiment 1, wherein generating thestabilized version of the second frame includes zooming into a versionof the second frame that was generated by applying the secondmathematical transformation to the second frame.

Embodiment 9 is the method of embodiment 1, wherein the operationsfurther comprise shifting a zoomed-in region of the version of thesecond frame horizontally or vertically to avoid the zoomed-in region ofthe second frame from presenting an invalid region.

Embodiment 10 is directed to a system including a recordable mediahaving instructions stored thereon, the instructions, when executed byone or more processors, cause performance of operations according to themethod of any one of embodiments 1 through 9.

Particular implementations can, in certain instances, realize one ormore of the following advantages. Video stabilization techniquesdescribed herein can compensate for movement in more than two degrees offreedom (e.g., more than just horizontal and vertical movement), forexample, by compensating for movement in eight degrees of freedom (e.g.,translation aspects, rotation aspects, zooming aspects, and non-rigidrolling shutter distortion). The video stabilization techniquesdescribed herein can operate as video is being captured by a device andmay not require information from future frames. In other words, thevideo stabilization techniques may be able to use information from onlypast frames in stabilizing a most-recently-recorded frame, so that thesystem can store a stabilized video stream as that video stream iscaptured (e.g., without storing multiple unstabilized video frames, suchas without storing more than 1, 100, 500, 1000, or 5000 unstabilizedvideo frames in a video that is being currently recorded or that hasbeen recorded). Accordingly, the system may not need to wait tostabilize a video until after the entire video has been recorded. Thedescribed video stabilization techniques may have low complexity, andtherefore may be able to run on devices that have modest processingpower (e.g., some smartphones). Moreover, the video stabilizationtechniques described herein may be able to operate in situations inwhich the frame-to-frame motion estimation fails in the first step.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features,objects, and advantages will be apparent from the description anddrawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 shows a diagram of a video stream that is being stabilized by avideo stabilization process.

FIGS. 2A-2B show a flowchart of a process for stabilizing video.

FIG. 3 is a block diagram of computing devices that may be used toimplement the systems and methods described in this document, as eithera client or as a server or plurality of servers.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

This document generally describes stabilizing video. The videostabilization may be performed by identifying a transformation between amost-recently-received frame of video and a previously-received frame ofa video (where the transformation indicates movement of the camera withrespect to a scene from frame to frame), modifying that transformationbased on information from past frames, generating a secondtransformation based on the transformation and the modifiedtransformation, and applying the second transformation to thecurrently-received frame to generate a stabilized version of thecurrently-received frame. This process is described generally withrespect to FIG. 1, and then with greater detail with respect to FIG. 2.

FIG. 1 shows a diagram of a video stream that is being stabilized by avideo stabilization process. The figure includes three frames of a video110 a-c. These frames may be in succession, such that frame 110 b may bethe frame that was immediately captured after frame 110 a was captured,and frame 110 c may be the frame that was immediately captured afterframe 110 b was captured. This document may occasionally refer to twoframes of a video as a first frame of a video and second frame of avideo, but the “first” notation does not necessarily mean that the firstframe is the initial frame in the entire video.

Frames 110 a-c are shown positioned between or near lines 112 a-b, whichindicate a position of the scenes represented by the frames with respectto each other. The lines are provided in this figure to show that thecamera was moving when it captured the frames 110 a-c. For example, thecamera was pointing more downwards when it captured frame 110 b thanwhen it captured frame 110 a, and was pointing more upwards when itcaptured frame 110 c than when it captured frames 110 a-b.

A computing system identifies a mathematical transformation (box 120)that indicates movement of the camera from the first frame 110 b to thesecond frame 110 c. The identification may be performed using frames 110b-c (as illustrated by the arrows in the figure), where frame 110 c maybe the most-recently-captured frame. These two frames 110 b-c may bereceived from a camera sensor or camera module that is attached to thecomputing system, or may be received from a remote device that capturedthe video frames 110 b-c. The identification of the mathematicaltransformation may include generating the mathematical transformation.The mathematical transformation may be a homography transform matrix, asdescribed in additional detail with respect to box 210 in FIGS. 2A-B.

A computing system then creates a modified transformation (box 125) bymodifying the initial transformation (box 120) so that the modifiedtransformation (box 125) is less representative than the initialtransformation of movement that began recently. The modifiedtransformation may be a low-pass filtered version of the initialtransformation. Doing so results in a modified mathematicaltransformation (box 125) that is more representative than the initialtransformation (box 120) of movement that has been occurring over a longperiod of time, in contrast to movement that began recently.

As an example, the modified mathematical transformation (box 125) maymore heavily represent a panning motion that has been occurring formultiple seconds than an oscillation that began a fraction of a secondago. Modifying the transformation in this manner takes into accountprevious frames of the video, as illustrated by the arrows in FIG. 1that point from frames 110 a-b to box 122. For example, transformationscalculated using the previous frames can be used to identify whichmovements have been occurring for a longer period of time, and whichmovements just began recently. An example way to use previous frames tocalculate the modified transformation can be to apply a lowpass filterto the homography transform matrix, as described in additional detailwith respect to box 220 in FIGS. 2A-B.

Box 130 shows a second transformation that is generated from the initialtransformation (box 120) and the modified transformation (box 125). Thesecond transformation may be a difference between the initialtransformation and the modified transformation. Generating the secondtransformation from the initial transformation and the modifiedtransformation is described in additional detail with respect to box 230in FIGS. 2A-B.

Box 132 shows how a computing system identifies an anticipateddistortion in a stabilized version of the second frame 110 c that wouldresult from applying the second transformation to the second frame 110c, based on a difference between (i) an amount of distortion in thehorizontal direction that would result from applying the secondmathematical transformation to the second frame 110 c, and (ii) anamount of distortion in the vertical direction that would result fromapplying the second mathematical transformation to the second frame 110c. Calculating the anticipated distortion is described in additionaldetail with respect to box 250 in FIGS. 2A-B.

Box 134 shows how a computing system determines an amount by which toreduce a stabilizing effect that results from applying the secondmathematical transformation to the second frame 110 c, based on a degreeto which the anticipated distortion exceeds an acceptable change indistortion. The acceptable change in distortion may be calculated usingmultiple frames of the video that preceded the second frame 110 c, asillustrated by the arrows in FIG. 1 that point from frames 110 a-b tobox 134. As an example, the distortion in multiple previous frames maybe analyzed, and if the distortion that would result from stabilizingthe current frame 110 c deviates significantly from an amount by whichthe distortion is changing from frame to frame, the computing system mayreduce the stabilization of the current frame 110 c to keep thedistortion from being too apparent to a viewer of the video. Determiningthe amount by which to reduce the video stabilization is described inadditional detail with respect to boxes 250 and 260 in FIG. 2. The useby the computing system of the determined amount by which to reduce thevideo stabilization may include generating a modified secondtransformation (box 140) using the determined amount.

The computing system generates the stabilized version of the secondframe 110 c (box 150) by applying the modified second transformation(box 140) to the second frame 110 c. Since the modified secondtransformation (box 140) has been modified based on the determinedamount by which to reduce the stabilization, the computing system'sgeneration of the stabilized version of the second frame (box 150) isconsidered to have been reduced based on the determined amount by whichto reduce the stabilization effect.

In some implementations, determining the amount by which to reduce thestabilizing effect is further or alternatively based on a determinedspeed of movement of the camera with respect to the scene from the firstframe 110 b to the second frame 110 c exceeding an acceptable change inspeed of the camera with respect to the scene. The acceptable change inspeed of the camera may be calculated from multiple frames of the videothat preceded the second frame 110 c, as described in greater detailwith respect to boxes 240 and 260 in FIGS. 2A-B.

In some implementations, generating the stabilized version of the secondframe includes zooming into a version of the second frame that isgenerated by applying the second mathematical transformation to thesecond frame. The computing system can shift a zoomed-in regionhorizontally, vertically, or both to avoid the zoomed-in region frompresenting an invalid region that may appear at the edges of thestabilized second frame. Doing so is described in greater detail withrespect to boxes 280 and 290 in FIGS. 2A-B.

FIGS. 2A-B show a flowchart of a process for stabilizing video. Thisprocess is represented by boxes 210 through 290, which are describedbelow. The operations described in association with those boxes may nothave to be performed in the order listed below or shown in FIGS. 2A-B.

At box 210, the computing system estimates a matrix that represents theframe-to-frame motion (“H_interframe”) using two video frames as input.This frame-to-frame motion matrix may be a homography transform matrix.A homography transform matrix may be a matrix that can represent themovement of a scene or a camera that was capturing a scene between twoframes of a video. As an example, each frame of a video may display atwo-dimensional image. Suppose that a first frame took a picture of asquare from straight in front of the square, so that the square hadequal-length sides with ninety-degree angles in the video frame (inother words it appeared square). Suppose now that the camera was movedto the side (or the square itself was moved) so that a next frame of thevideo displayed the square as skewed with some sides longer than eachother and with angles that are not ninety degrees. The location of thefour corner points of the square in the first frame can be mapped to thelocation of the four corner points in the second frame to identify howthe camera or scene moved from one frame to the next.

The mapping of these corner points to each other in the frames can beused to generate a homography transform matrix that represents themotion of the camera viewpoint with respect to the scene that it isrecording. Given such a homography transform matrix, a first frame canbe used with the generated homography transform matrix to recreate thesecond frame, for example, by moving pixels in the first frame todifferent locations according to known homography transformationmethods.

The homography transform matrix that is described above can representnot only translational movement, but also rotation, zooming, andnon-rigid rolling shutter distortion. In this way, application of thehomography transform matrix can be used to stabilize the video withrespect to movement in eight degrees-of-freedom. To compare, some videostabilization mechanisms only stabilize images to account fortranslational movement (e.g., up/down and left/right movement).

The above-described homography transform matrix may be a 3×3 homographytransform matrix, although other types of homography matrices may beused (and other mathematical representations of movement from one frameto another, even if not a homography matrix or even if not a matrix, maybe used). The 3×3 matrix (referred to as H_interface) may be determinedin the following manner. First, a computing system finds a set offeature points (usually corner points) in the current image, where thosepoints are denoted [x′_i, y′_i], i=1 . . . N (N is the number of featurepoints). Then, corresponding feature points in the previous frame arefound, where the corresponding feature points are denoted [x_i, y_i].Note that the points are described as being in the GL coordinate system(i.e., the x and y ranges from −1 to 1 and with the frame center as theorigin). If the points are in the image pixel coordinate system in whichx ranges from 0 to the image width and y ranges from 0 to the imageheight, then the points can be transformed to the GL coordinate systemor the resulting matrix can be transformed to compensate.

The above-described H_interfame matrix is a 3×3 matrix which contains 9elements:

${H\_ interframe} = \begin{matrix}{{h\; 1},{h\; 2},{h\; 3}} \\{{h\; 4},{h\; 5},{h\; 6}} \\{{h\; 7},{h\; 8},{h\; 9}}\end{matrix}$H_interfame is the transform matrix that transforms [x_i, y_i] into[x′_i, y′_i], as described below.z_i′*[x′_i,y′_i,1]′=H_interframe*[x_i,y_i,1]′[x′_i, y′_i, 1]′ is a 3×1 vector which is the transpose of [x′_i, y′_i,1] vector. [x_i, y_i, 1]′ is a 3×1 vector which is the transpose of[x_i, y_i, 1] vector. z_i′ is a scale factor.

Given a set of corresponding feature points, an example algorithm forestimating the matrix is described in the following computer vision bookat algorithm 4.1 (page 91) and at algorithm 4.6 (page 123): “Hartley,R., Zisserman, A.: Multiple View Geometry in Computer Vision. CambridgeUniversity Press (2000),” available atftp://vista.eng.tau.ac.il/dropbox/aviad/Hartley,%20Zisserman%20-%20Multiple%20View%20Geometry%20in%20Computer%20Vision.pdf

At box 220, the computing system estimates a lowpass transform matrix(H_lowpass). The lowpass transform matrix may later be combined with theH_interframe matrix to generate a new matrix (H_compensation) that canbe used to remove the results of involuntary “high frequency” movementof the video camera. If the system attempted to remove all movement (inother words, did not do the lowpass filtering described herein), theuser may not be able to move the camera voluntarily and have the scenedepicted by the video also move. As such, the computing systemsgenerates the lowpass transformation in order to filter out highfrequency movements. High frequency movements may be those movementsthat are irregular and that are not represented through many frames,such as back and forth movements with a short period. To the contrary,low frequency movements may be those movements that are representedthrough many frames, such a user panning a video camera for multipleseconds.

To perform this filtering, the computing system generates a lowpasstransform matrix (H_lowpass) that includes values weighted to emphasizethe low frequency movements that have been occurring over a long timeseries. The lowpass transform matrix may be the result of applying alow-pass filter to the H_interframe matrix. Each element in the lowpasstransform matrix is generated individually, on an element-by-elementbasis, from (1) its own time series of the lowpass transform matrix fromthe previous frame, (2) the H_interframe matrix that represents movementbetween the previous frame and the current frame, and (3) a dampeningratio that is specified by the user. In other words, the elements in thematrix that are weighted with notable values may be those elements thatrepresent movement that has been present in the H_interframe matrixthrough many frames. The equation to generate H_lowpass may berepresented as follows:H_lowpass=H_previous_lowapss*transform_damping_ratio+H_interframe*(1−transform_damping_ratio)This equation is an example of a two-tap infinite impulse responsefilter.

At box 230, the computing system computes a compensation transformmatrix (H_compensation). The compensation matrix may be a combination ofthe lowpass matrix (H_lowpass) and the frame-to-frame motion matrix(H_interframe). Combining these two matrices generates a matrix(H_compensation) that is needed to keep the movement from one frame toanother, but only those movements that have been occurring for areasonable period of time, to the exclusion of recent “involuntary”movements. The H_compensation matrix may represent the difference inmovements between H_interfame and the H_lowpass, such that whileH_lowpass could be applied to the last frame to generate a modifiedversion of the last frame that represents the voluntary movements thatoccurred between the last frame to the current frame, H_compensation maybe applied to the current frame to generate a modified (and stabilized)version of the current frame that represents the voluntary movementsthat occurred between the last frame and the current frame. In roughlanguage, applying H_compensation the current frame removes theinvoluntary movement from that frame. Specifically, given this computedH_compensation matrix, the system should be able to take the currentframe of the video, apply the H_compensation matrix to that frame with atransformation process, and obtain a newly-generated frame that issimilar to the current frame, but that excludes any such sudden andsmall movement. In other words, the system attempts to keep the currentframe as close as possible to the last frame, but permits long-term“voluntary” movements.

The compensation transform matrix may be generated with the followingequation:H_compensation=Normalize(Normalize(Normalize(H_lowpass)*H_previous_compensation)*Inverse(H_interframe))The H_previous_compensation matrix is the H_constrained_compensationmatrix that is calculated later on this process, but that which wascalculated for the previous frame. Inverse( ) is the matrix inverseoperation that is used to generate the original version of the lastframe by inversing the transformation. Combining the original version ofthe last frame with the lowpass filter matrix permits the voluntarymovements. Combining with H_previous_compensation compensates for theprevious compensation value.

Normalize( ) is an operation that normalizes the 3×3 matrix by itssecond singular value. The normalization process is performed becausesome of the steps of the process may result in a transformation that,for lack of better words, would not make much sense in the real world.As such, a normalization process can make sure that a reasonable resultis being obtained from each process step. The normalization is performedfor each step of the process, so that an odd output from one step doesnot pollute the remaining steps of the process (e.g., imagine if the oddoutput provided a near-zero value that would pull the output of the restof the steps to also be near-zero). For reasons that are discussedbelow, additional processing may enhance the results of the videostabilization process.

At box 240, the computing system computes a speed reduction value. Thespeed reduction value may be a value that is used to determine how muchto reduce video stabilization when the camera moves very quickly andvideo stabilization becomes unwelcome because frame-to-frame motion maybe unreliable. To calculate the amount by which the video stabilizationmay be reduced, the speed of movement between frames is initiallycalculated. In this example, the computing system generates the speed ofthe center of the frame. The speed in the x direction is pulled from therow 1 column 3 element in the H_interframe as follows (box 242):speed_x=H_interframe[1,3]*aspect_ratioThe speed in the y direction is pulled from the row 2, column 3 elementin the H_interfarme matrix, as follows (box 242):speed_y=H_interframe[2,3]The above-described aspect_ratio is the frame_width divided by theframe_height. These identifications of speed may only account for thetranslation movement between two frames, but speed may account forrotation, zooming, or other types of movements, in other examples.

The system may then determine the lowpass motion speed that accounts forthe long-term speed of the camera (or the scene) and excludes sudden andquick “involuntary” movements. This is done by taking the current speedand combining it with the previously-calculated lowpass speed, and byfurther applying a damping ratio that inversely weights the currentspeed with respect to the previously-calculated lowpass speed, forexample, as follows:lowpass_speed_x=lowpass_speed_x_previous*speed_damping_ratio+speed_x*(1−speed_damping_ratio)This equation in effect generates a lowpass speed by taking thepreviously-calculated speed and reducing it by an amount that isspecified by a damping ratio. The reduction is compensated by thecurrent speed. In this way, the current speed of the video affects theoverall lowpass_speed value, but is not an exclusive factor in thelowpass_speed value. The above equation represents an infinite impulseresponse filter. The same process may be performed for the y speed togenerate the lowpass y speed, for example, as follows:lowpass_speed_y=lowpass_speed_y_previous*speed_damping_ratio+speed_y*(1−speed_damping_ratio)The damping ratio in this process is set by a user, and an example valueis 0.99.

The process then combines these values to generate a singlerepresentation of the lowpass speed that accounts for movement in the xand y directions, for example, using the following equation (box 244):lowpass_speed=sqrt(lowpass_speed_x*lowpass_speed_x+lowpass_speed_y*lowpass_speed_y)This calculated lowpass speed essentially represents the long-term speedof movement between frames. In other words, lowpass_speed accounts lessfor recent changes in speed and weights more heavily the longer-termspeed trend.

With the lowpass speed calculated, the system can calculate the speedreduction value. In some examples, the speed reduction value is a valuebetween 0 and 1 (other boundary values are possible), and the system maygenerate the speed reduction value based on how the lowpass speedcompares to a low threshold and a high threshold. If the lowpass speedis below the low threshold, the speed reduction value may be set to the0 boundary value. If the lowpass speed is above the high threshold, thespeed reduction value may be set to the 1 boundary value. If the lowpassspeed is between the two thresholds, the computing system may select aspeed reduction value that represents the lowpass speed's scaled valuebetween the thresholds, e.g., where the speed reduction value liesbetween the 0 and 1 boundary values. The calculation of the speedrejection value can be represented with the following algorithm (box246):If lowpass_speed<low_speed_threshold, then speed_reduction=0Else if lowpass_speed>high_speed_threshold, thenspeed_reduction=max_speed_reductionOtherwise,speed_reduction=max_speed_reduction*(lowpass_speed−low_speed_threshold)/(high_speed_threshold−low_speed_threshold)With this algorithm, the low_speed_threshold, high_speed_threshold, andmax_speed_reduction are all specified by the user. Example valuesinclude low_speed_threshold=0.008; high_speed_threshold=0.016; andmax_speed_reduction=1.0.

At box 250, the computing system calculates a distortion reductionvalue. The computing system may calculate a distortion reduction valuebecause the compensation transform may create too much non-rigiddistortion when applied to a video frame. In other words, the videostabilization may not appear realistic, for example, because thedistortion caused by stretching the image in one direction more thananother may occur too quickly and could appear unusual to a user.

To calculate the distortion reduction value, the computing system mayfirst compute the compensation zoom factor by looking to the values forthe zoom factors in the H_compensation matrix, as follows:zoom_x=H_compensation[1,1] which is the row 1 column 1 element in theH_compensation matrixzoom_y=H_compensation[2,2] which is the row 2 column 2 element in theH_compensation matrixThe zoom factors may be those factors that identify how thetransformation stretches the image in a dimension.

The computing system may then determine the difference between the twozoom factors, to determine the degree to which the image is beingdistorted by stretching more in one direction than the other, as follows(box 252):distortion=abs(zoom_x−zoom_y)

A lowpass filter is applied to the distortion, to place a limit on therate at which the distortion is permitted to change and therefore tomake sure that sudden changes in distortion minimized, using thefollowing formula (box 254):lowpass_distortion=previous_lowpass_distortion*distortion_damping_ratio+distortion*(1−distortion_damping_ratio)Stated another way, the algorithm is arranged to allow the amount ofdistortion to change slowly. In the above formula, thedistortion_damping_ratio is the damping ratio for the distortion IIRfilter that is specified by the user. An example value is 0.99.

With the lowpass distortion calculated, the computing system cancalculate the distortion reduction value. In some examples, thedistortion reduction value is a value between 0 and 1 (other boundaryvalues are possible), and the system may generate the distortionreduction value based on how the lowpass distortion compares to a lowthreshold and a high threshold. If the lowpass distortion is below thelow threshold, the lowpass reduction value may be set to the 0 boundaryvalue. If the lowpass distortion value is above the high threshold, thedistortion reduction value may be set to the 1 boundary value. If thelowpass distortion value is between the two thresholds, a value may beselected that represents the lowpass distortion's scaled value betweenthe thresholds (e.g., where the resulting distortion reduction valuelies between the 0 and 1 boundary values). The calculation of thedistortion reduction value can be represented with the followingalgorithm (box 256):If lowpass_distortion<low_distortion_threshold, thendistortion_reduction=0Else if lowpass_distortion>high_distortion_threshold, thenmax_distortion_reductionOtherwise,distortion_reduction=max_distortion_reduction*(lowpass_distortion−low_distortion_threshold)/(high_distortion_threshold−low_distortion_threshold)With this algorithm, low_distortion_threshold,high_distortion_threshold, and max_distortion_reduction are allspecified by the user. Example values includelow_distortion_threshold=0.001, high_distortion_threshold=0.01, andmax_distortion_reduction=0.3.

At box 260, the computing system reduces a strength of the videostabilization based on the determined speed reduction value anddistortion reduction value. To do this, the computing system calculatesa reduction value, which in this example is identified as a maximum ofthe speed reduction value and the distortion reduction value (box 262),as follows:reduction=max(speed_reduction, distortion_reduction)In other examples, the reduction value may be a combination of these twovalues that accounts for a portion of each value (e.g., the values maybe added or multiplied together, and possibly then multiplied by apredetermined number such as 0.5). The reduction value may fall at orbetween the boundary values of 0 and 1, and the closer that thereduction value is to 1, the more that the computing system may reducethe strength of the image stabilization.

The computing system may then modify the compensation transform matrixto generate a reduced compensation transform matrix (box 264). Thecomputing system may do this by multiplying the compensation transformmatrix by a subtraction of the reduction value from one. In other words,if the reduction value is very near one (indicating that there is to bea great reduction in image stabilization), the values in thecompensation matrix may be significantly diminished because they wouldbe multiplied by a number near zero. The numbers in the modifiedcompensation transform matrix are then added to an identity matrix thathas been multiplied by the reduction value. An example equation follows:H_reduced_compensation=Identity*reduction+H_compensation*(1−reduction)

At box 270, the computing system may constrain the compensation so thatresulting video stabilization does not show invalid regions of theoutput frame (e.g., those regions that are outside of the frame). Assome background, because the compensation may warp the image thatresults from the image stabilization process, that image may displayinvalid regions that are essentially outside of the image at itsborders. To make sure that these invalid regions are not shown, thecomputing system may zoom into the image to crop out sides of the imagethat could include the invalid regions.

Getting back to the processes of box 270, if the camera is moved quicklyand significantly, the stabilization could lock into displaying an oldlocation because the quick and significant movement may be filtered out,which may introduce the above-described invalid regions into the displayof a stabilized frame. In such a case, the below-described constraintprocess can ensure that the video stabilization essentially stopsfully-controlling the region of the frame that is displayed if the videowould be about to display an invalid region. This determinationregarding whether the stabilization needs to give up some control of theframe may start by initially setting the corner points of the outputimage and determining if those corner points fall outside of apre-specified cropping region. A maximum amount of compensation andzooming may be defined as twice the cropping ratio, where the croppingratio may be specified by a user (e.g., to be 15% on each side, or 0.15in the below equation):max_compensation=cropping_ratio*2

The computing system may then use the H_reduced_compensation matrix totransform the 4 corners of a unit square in GL coordinate (x01,y01)=(−1, −1), (x02, y02)=(1, −1), (x03, y03)=(−1,1), (x04, y04)=(1,1)to the 4 corner points (x1, y1), (x2, y2), (x3, y3), (x4, y4). (Notethat the video frame does not need to be a unit square, but thedimensions of the unit frame are mapped to a unit square in GLcoordinate). More specifically, we use the following formula totransform (x0i, y0i) into (xi, yi):dzi*[xi,yi,1]′=H_reduced compensation*[x0i,y0i,1]′In this example, [x0i, y0i, 1]′ is a 3×1 vector which is the transposeof the [x0i, y0i, 1] vector. [xi, yi, 1]′ is a 3×1 vector which is thetranspose of [xi, yi, 1] vector. zi is a scale factor.

The computing system may then identify the maximum amount ofdisplacement in each direction (left, right, up, and down) from thecorners of each transformed video frame to the edge of the unit square,as follows:max_left_displacement=1+max(x1,x3)max_right_displacement=1−min(x2,x4)max_top_displacement=1+max(y1,y2)max_bottom_displacement=1−min(y3,y4)If any of the identified displacements exceed the maximum amount ofcompensation (which is twice the cropping ratio as described above, andwhich would indicate that the invalid regions are in the displayedregion of the zoomed-in region of the unit square), then the cornerpoints of the frame are shifted by a same amount away from the edge ofthe unit square so that the invalid regions will not be displayed.Equations for shifting the corner points accordingly follow:If max_left_displacement>max_compensation, shift the 4 corner pointsleft by max_left_displacement−max_compensationIf max_right_displacement>max_compensation, shift the 4 corner pointsright by max_right_displacement−max_compensationIf max_top_displacement>max_compensation, shift the 4 corner points upby max_top_displacement−max_compensationIf max_bottom_displacement>max_compensation, shift the 4 corner pointsdown by max_bottom_displacement−max_compensation.Shifting the corner points is an identification that invalid regionswould have been shown even if the display was cropped (box 272).

After all of the above shifting operations, the 4 new corner points maybe denoted (x1′, y1′), (x2′, y2′), (x3′, y3′), (x4′, y4′). The computingsystem then computes the constrained compensation transform matrixH_constrained_compensation, which maps the four corners of a unit squarein GL coordinate (x01, y01)=(−1, −1), (x02, y02)=(1, −1), (x03,y03)=(−1,1), (x04, y04)=(1,1) to the 4 constrained corner points (x1′,y1′), (x2′, y2′), (x3′, y3′), (x4′, y4′), as follows:zi′*[xi′,yi′,1]′=H_constrained_compensation*[x0i,y0i,1]′In this example, [x0i, y0i, 1]′ is a 3×1 vector which is the transposeof [x0i, y0i, 1] vector. [xi′, yi′, 1]′ is a 3×1 vector which is thetranspose of [xi′, yi′, 1] vector. zi′ is a scale factor. Given 4 pairsof points [x0i, y0i, 1]′ and [xi′, yi′, 1]′, an example algorithm forestimating the matrix is described in the following computer vision bookat algorithm 4.1 (page 91): “Hartley, R., Zisserman, A.: Multiple ViewGeometry in Computer Vision. Cambridge University Press (2000),”available atftp://vista.eng.tau.ac.il/dropbox/aviad/Hartley,%20Zisserman%20-%20Multiple %20View %20Geometry %20in %20Computer %20Vision. pdf

The H_constrained_compensation is then saved as H_previous_compensation,which may be used in computations for stabilizing the next frame, asdescribed above with respect to box 230.

At box 280, the computing system modifies the constrained compensationmatrix so that the stabilized image will be zoomed to crop the border.In some examples, the computing system first identifies a zoom factor asfollows:zoom_factor=1/(1−2*cropping_ratio)In doing so, the computing system doubles the crop ratio (e.g., bydoubling the 15% value of 0.15 to 0.3), subtracting the resulting valuefrom 1 (e.g., to get 0.7), and then dividing 1 by that result to get thezoom factor (e.g., 1 divided by 0.7 to equal a 1.42 zoom factor). Thecomputing system may then divide certain features of the the constrainedcompensation matrix in order to zoom into the display a certain amount,as follows:H_constrained_compensation[3,1]=H_constrained_compensation[3,1]/zoom_factorH_constrained_compensation[3,2]=H_constrained_compensation[3,2]/zoom_factorH_constrained_compensation[3,3]=H_constrained_compensation[3,3]/zoom_factor

At box 290, the computing system applies the modified constrainedcompensation matrix to the current frame in order to generate a croppedand stabilized version of the current frame. An example way to apply theconstrained compensation matrix (H_constrained_compensation) on theinput frame to produce the output image can be described as follows.z′*[x′,y′,1]′=H_constrained_compensation*[x,y,1]′

-   -   [x, y, 1]′ is a 3×1 vector that represents a coordinate in the        input frame    -   [x′, y′, 1]′ is a 3×1 vector that represents the coordinate in        the output frame    -   z′ is a scale factor    -   H_constrained_compensation is a 3×3 matrix which contains 9        elements:

${{H\_ constrained}{\_ compensation}} = \begin{matrix}{{h\; 1},{h\; 2},{h\; 3}} \\{{h\; 4},{h\; 5},{h\; 6}} \\{{h\; 7},{h\; 8},{h\; 9}}\end{matrix}$

In additional detail, For each pixel [x, y] in the input frame, find theposition [x′, y′] in the output frame using the above transformation,and copy the pixel value from [x, y] in the input frame to [x′, y′] inthe output frame. Another way is for each pixel [x′, y′] in the outputframe, find the position [x, y] in the input frame using the inversetransformation, and copy the pixel value from [x, y] in the input imageto [x′, y′] in the output frame. These operations may be performed inthe computing systems Graphics Processing Unit (GPT) efficiently.

The process described herein for boxes 210 through 290 may then repeatedfor the next frame, with some of the values from processing the currentframe being used for the next frame.

In various implementations, operations that are performed “in responseto” or “as a consequence of” another operation (e.g., a determination oran identification) are not performed if the prior operation isunsuccessful (e.g., if the determination was not performed). Operationsthat are performed “automatically” are operations that are performedwithout user intervention (e.g., intervening user input). Features inthis document that are described with conditional language may describeimplementations that are optional. In some examples, “transmitting” froma first device to a second device includes the first device placing datainto a network for receipt by the second device, but may not include thesecond device receiving the data. Conversely, “receiving” from a firstdevice may include receiving the data from a network, but may notinclude the first device transmitting the data.

“Determining” by a computing system can include the computing systemrequesting that another device perform the determination and supply theresults to the computing system. Moreover, “displaying” or “presenting”by a computing system can include the computing system sending data forcausing another device to display or present the referenced information.

In various implementations, operations that are described as beingperformed on a matrix means operations that are performed on that matrixor a version of that matrix that has been modified by an operationdescribed in this disclosure or an equivalent thereof.

FIG. 3 is a block diagram of computing devices 300, 350 that may be usedto implement the systems and methods described in this document, aseither a client or as a server or plurality of servers. Computing device300 is intended to represent various forms of digital computers, such aslaptops, desktops, workstations, personal digital assistants, servers,blade servers, mainframes, and other appropriate computers. Computingdevice 350 is intended to represent various forms of mobile devices,such as personal digital assistants, cellular telephones, smartphones,and other similar computing devices. The components shown here, theirconnections and relationships, and their functions, are meant to beexamples only, and are not meant to limit implementations describedand/or claimed in this document.

Computing device 300 includes a processor 302, memory 304, a storagedevice 306, a high-speed interface 308 connecting to memory 304 andhigh-speed expansion ports 310, and a low speed interface 312 connectingto low speed bus 314 and storage device 306. Each of the components 302,304, 306, 308, 310, and 312, are interconnected using various busses,and may be mounted on a common motherboard or in other manners asappropriate. The processor 302 can process instructions for executionwithin the computing device 300, including instructions stored in thememory 304 or on the storage device 306 to display graphical informationfor a GUI on an external input/output device, such as display 316coupled to high-speed interface 308. In other implementations, multipleprocessors and/or multiple buses may be used, as appropriate, along withmultiple memories and types of memory. Also, multiple computing devices300 may be connected, with each device providing portions of thenecessary operations (e.g., as a server bank, a group of blade servers,or a multi-processor system).

The memory 304 stores information within the computing device 300. Inone implementation, the memory 304 is a volatile memory unit or units.In another implementation, the memory 304 is a non-volatile memory unitor units. The memory 304 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 306 is capable of providing mass storage for thecomputing device 300. In one implementation, the storage device 306 maybe or contain a computer-readable medium, such as a floppy disk device,a hard disk device, an optical disk device, or a tape device, a flashmemory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The information carrier is a computer- ormachine-readable medium, such as the memory 304, the storage device 306,or memory on processor 302.

The high-speed controller 308 manages bandwidth-intensive operations forthe computing device 300, while the low speed controller 312 manageslower bandwidth-intensive operations. Such allocation of functions is anexample only. In one implementation, the high-speed controller 308 iscoupled to memory 304, display 316 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 310, which may acceptvarious expansion cards (not shown). In the implementation, low-speedcontroller 312 is coupled to storage device 306 and low-speed expansionport 314. The low-speed expansion port, which may include variouscommunication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)may be coupled to one or more input/output devices, such as a keyboard,a pointing device, a scanner, or a networking device such as a switch orrouter, e.g., through a network adapter.

The computing device 300 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 320, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 324. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 322. Alternatively, components from computing device 300 may becombined with other components in a mobile device (not shown), such asdevice 350. Each of such devices may contain one or more of computingdevice 300, 350, and an entire system may be made up of multiplecomputing devices 300, 350 communicating with each other.

Computing device 350 includes a processor 352, memory 364, aninput/output device such as a display 354, a communication interface366, and a transceiver 368, among other components. The device 350 mayalso be provided with a storage device, such as a microdrive or otherdevice, to provide additional storage. Each of the components 350, 352,364, 354, 366, and 368, are interconnected using various buses, andseveral of the components may be mounted on a common motherboard or inother manners as appropriate.

The processor 352 can execute instructions within the computing device350, including instructions stored in the memory 364. The processor maybe implemented as a chipset of chips that include separate and multipleanalog and digital processors. Additionally, the processor may beimplemented using any of a number of architectures. For example, theprocessor may be a CISC (Complex Instruction Set Computers) processor, aRISC (Reduced Instruction Set Computer) processor, or a MISC (MinimalInstruction Set Computer) processor. The processor may provide, forexample, for coordination of the other components of the device 350,such as control of user interfaces, applications run by device 350, andwireless communication by device 350.

Processor 352 may communicate with a user through control interface 358and display interface 356 coupled to a display 354. The display 354 maybe, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display)display or an OLED (Organic Light Emitting Diode) display, or otherappropriate display technology. The display interface 356 may compriseappropriate circuitry for driving the display 354 to present graphicaland other information to a user. The control interface 358 may receivecommands from a user and convert them for submission to the processor352. In addition, an external interface 362 may be provide incommunication with processor 352, so as to enable near areacommunication of device 350 with other devices. External interface 362may provided, for example, for wired communication in someimplementations, or for wireless communication in other implementations,and multiple interfaces may also be used.

The memory 364 stores information within the computing device 350. Thememory 364 can be implemented as one or more of a computer-readablemedium or media, a volatile memory unit or units, or a non-volatilememory unit or units. Expansion memory 374 may also be provided andconnected to device 350 through expansion interface 372, which mayinclude, for example, a SIMM (Single In Line Memory Module) cardinterface. Such expansion memory 374 may provide extra storage space fordevice 350, or may also store applications or other information fordevice 350. Specifically, expansion memory 374 may include instructionsto carry out or supplement the processes described above, and mayinclude secure information also. Thus, for example, expansion memory 374may be provide as a security module for device 350, and may beprogrammed with instructions that permit secure use of device 350. Inaddition, secure applications may be provided via the SIMM cards, alongwith additional information, such as placing identifying information onthe SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 364, expansionmemory 374, or memory on processor 352 that may be received, forexample, over transceiver 368 or external interface 362.

Device 350 may communicate wirelessly through communication interface366, which may include digital signal processing circuitry wherenecessary. Communication interface 366 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 368. In addition, short-range communication may occur, suchas using a Bluetooth, WiFi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module 370 mayprovide additional navigation- and location-related wireless data todevice 350, which may be used as appropriate by applications running ondevice 350.

Device 350 may also communicate audibly using audio codec 360, which mayreceive spoken information from a user and convert it to usable digitalinformation. Audio codec 360 may likewise generate audible sound for auser, such as through a speaker, e.g., in a handset of device 350. Suchsound may include sound from voice telephone calls, may include recordedsound (e.g., voice messages, music files, etc.) and may also includesound generated by applications operating on device 350.

The computing device 350 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 380. It may also be implemented as part of asmartphone 382, personal digital assistant, or other similar mobiledevice.

Additionally computing device 300 or 350 can include Universal SerialBus (USB) flash drives. The USB flash drives may store operating systemsand other applications. The USB flash drives can include input/outputcomponents, such as a wireless transmitter or USB connector that may beinserted into a USB port of another computing device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), peer-to-peernetworks (having ad-hoc or static members), grid computinginfrastructures, and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Although a few implementations have been described in detail above,other modifications are possible. Moreover, other mechanisms forperforming the systems and methods described in this document may beused. In addition, the logic flows depicted in the figures do notrequire the particular order shown, or sequential order, to achievedesirable results. Other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method, comprising:receiving, by a computing system, first and second frames of a videothat was captured by a camera; generating, by the computing system andusing the first and second frames of the video, a first mathematicaltransformation that indicates movement of the camera with respect to ascene captured by the video from when the first frame was captured towhen the second frame was captured, including movement that beganrecently and movement that has been occurring over a longer term, thecomputing system generating the first mathematical transformation byidentifying how features points that are present in the first frame andthe second frame moved between the first frame and the second frame;identifying, by the computing system, movement of the camera withrespect to the scene that has been occurring over the longer term;generating, by the computing system using the first mathematicaltransformation and the identified movement of the camera with respect tothe scene that has been occurring over the longer term, a secondmathematical transformation by a process that involves removing from thefirst mathematical transformation indications of the identified movementof the camera that has been occurring over the longer term, the secondmathematical transformation being more representative of movement of thecamera with respect to the scene that began recently and lessrepresentative of movement of the camera with respect to the scene thathas been occurring over the longer term, in comparison to the firstmathematical transformation; generating, by the computing system, astabilized version of the second frame by applying the secondmathematical transformation to the second frame to remove from thesecond frame movement that began recently and leave movement that hasbeen occurring over the longer term, without taking into accountmovement of the camera with respect to the scene from future frames ofthe video.
 2. The computer-implemented method of claim 1, wherein thesecond frame is a frame of the video that immediately follows the firstframe of the video.
 3. The computer-implemented method of claim 1,wherein the first mathematical transformation that indicates movement ofthe camera with respect to the scene comprises a homographytransformation matrix.
 4. The computer-implemented method of claim 1,wherein generating the stabilized version of the second frame includeszooming into a version of the second frame that was generated byapplying the second mathematical transformation to the second frame. 5.The computer-implemented method of claim 4, further comprising shiftinga zoomed-in region of the stabilized version of the second framehorizontally or vertically to avoid the zoomed-in region of thestabilized version of second frame from presenting an invalid region. 6.The computer-implemented method of claim 1, wherein identifying themovement of the camera that has been occurring over the longer termincludes analyzing frames of the video that the camera captured beforethe first and second frames of the video to identify movement that hasbeen occurring over the longer term.
 7. The computer-implemented methodof claim 1, wherein identifying the movement of the camera that has beenoccurring over the longer term includes applying a lowpass filter to thefirst mathematical transformation based on analysis of frames of thevideo that the camera captured before the first and second frames of thevideo.
 8. The computer-implemented method of claim 1, furthercomprising: identifying, by the computing system, an anticipateddistortion that would be present in a stabilized version of the secondframe resulting from application of the second mathematicaltransformation to the second frame, based on a difference between: (i)an amount of distortion in the horizontal direction resulting fromapplication of the second mathematical transformation to the secondframe, and (ii) an amount of distortion in the vertical directionresulting from application of the second mathematical transformation tothe second frame; determining, by the computing system, an amount bywhich to reduce a stabilizing effect that results from application ofthe second mathematical transformation to the second frame, based on adegree to which the anticipated distortion exceeds an acceptable changein distortion that was calculated from distortion in multiple frames ofthe video that preceded the second frame, wherein generating thestabilized version of the second frame includes reducing a stabilizingeffect of applying the second mathematical transformation to the secondframe based on the determined amount by which to reduce the stabilizingeffect.
 9. The computer-implemented method of claim 1, furthercomprising: calculating, by the computing system, an acceptable changein speed of movement of the camera based on analysis of a speed ofmovement of the camera between multiple frames of the video thatpreceded the second frame; and determining, by the computing system, anamount by which to reduce a stabilizing effect that results fromapplication of the second mathematical transformation to the secondframe, based on a determined speed of movement of the camera from thefirst frame to the second frame exceeding the acceptable change in speedof movement of the camera, wherein generating the stabilized version ofthe second frame includes reducing a stabilizing effect of applying thesecond mathematical transformation to the second frame based on thedetermined amount by which to reduce the stabilizing effect.
 10. One ormore non-transitory computer-readable devices including instructionsthat, when executed by one or more processors, cause performance ofoperations that include: receiving, by a computing system, first andsecond frames of a video that was captured by a camera; generating, bythe computing system and using the first and second frames of the video,a first mathematical transformation that indicates movement of thecamera with respect to a scene captured by the video from when the firstframe was captured to when the second frame was captured, includingmovement that began recently and movement that has been occurring over alonger term, the computing system generating the first mathematicaltransformation by identifying how features points that are present inthe first frame and the second frame moved between the first frame andthe second frame; identifying, by the computing system, movement of thecamera with respect to the scene that has been occurring over the longerterm; generating, by the computing system using the first mathematicaltransformation and the identified movement of the camera with respect tothe scene that has been occurring over the longer term, a secondmathematical transformation by a process that involves removing from thefirst mathematical transformation indications of the identified movementof the camera that has been occurring over the longer term, the secondmathematical transformation being more representative of movement of thecamera with respect to the scene that began recently and lessrepresentative of movement of the camera with respect to the scene thathas been occurring over the longer term, in comparison to the firstmathematical transformation; generating, by the computing system, astabilized version of the second frame by applying the secondmathematical transformation to the second frame to remove from thesecond frame movement that began recently and leave movement that hasbeen occurring over the longer term, without taking into accountmovement of the camera with respect to the scene from future frames ofthe video.
 11. The one or more non-transitory computer-readable devicesof claim 10, wherein the second frame is a frame of the video thatimmediately follows the first frame of the video.
 12. The one or morenon-transitory computer-readable devices of claim 10, wherein the firstmathematical transformation that indicates movement of the camera withrespect to the scene comprises a homography transformation matrix. 13.The one or more non-transitory computer-readable devices of claim 10,wherein generating the stabilized version of the second frame includeszooming into a version of the second frame that was generated byapplying the second mathematical transformation to the second frame. 14.The one or more non-transitory computer-readable devices of claim 13,wherein the operations further comprise shifting a zoomed-in region ofthe stabilized version of the second frame horizontally or vertically toavoid the zoomed-in region of the stabilized version of second framefrom presenting an invalid region.
 15. The one or more non-transitorycomputer-readable devices of claim 10, wherein identifying the movementof the camera that has been occurring over the longer term includesanalyzing frames of the video that the camera captured before the firstand second frames of the video to identify movement that has beenoccurring over the longer term.
 16. The one or more non-transitorycomputer-readable devices of claim 10, wherein identifying the movementof the camera that has been occurring over the longer term includesapplying a lowpass filter to the first mathematical transformation basedon analysis of frames of the video that the camera captured before thefirst and second frames of the video.
 17. The one or more non-transitorycomputer-readable devices of claim 10, wherein the operations furthercomprise: identifying, by the computing system, an anticipateddistortion that would be present in a stabilized version of the secondframe resulting from application of the second mathematicaltransformation to the second frame, based on a difference between: (i)an amount of distortion in the horizontal direction resulting fromapplication of the second mathematical transformation to the secondframe, and (ii) an amount of distortion in the vertical directionresulting from application of the second mathematical transformation tothe second frame; determining, by the computing system, an amount bywhich to reduce a stabilizing effect that results from application ofthe second mathematical transformation to the second frame, based on adegree to which the anticipated distortion exceeds an acceptable changein distortion that was calculated from distortion in multiple frames ofthe video that preceded the second frame, wherein generating thestabilized version of the second frame includes reducing a stabilizingeffect of applying the second mathematical transformation to the secondframe based on the determined amount by which to reduce the stabilizingeffect.
 18. The one or more non-transitory computer-readable devices ofclaim 10, wherein the operations further comprise: calculating, by thecomputing system, an acceptable change in speed of movement of thecamera based on analysis of a speed of movement of the camera betweenmultiple frames of the video that preceded the second frame; anddetermining, by the computing system, an amount by which to reduce astabilizing effect that results from application of the secondmathematical transformation to the second frame, based on a determinedspeed of movement of the camera from the first frame to the second frameexceeding the acceptable change in speed of movement of the camera,wherein generating the stabilized version of the second frame includesreducing a stabilizing effect of applying the second mathematicaltransformation to the second frame based on the determined amount bywhich to reduce the stabilizing effect.